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Abstract 

Sounds are an integral and sometimes annoying part of our daily life. Virtual 
worlds which imitate natural environments gain a lot of authenticity from fast, 
high quality visualization combined with sound effects. Sounds help to increase 
the degree of immersion for human dwellers in imaginary worlds significantly. 

The virtual reality toolkit of IGD features a broad range of standard visual and 
advanced realtime audio components which interprete an object-oriented 
definition of the scene. The virtual reality system "Virtual Design" realized with 
the toolkit enables the designer of virtual worlds to create a true audiovisual 
environment. Several examples on video demonstrate the usage of the audio 
features in Virtual Design. 


1 INTRODUCTION 

Virtual reality applications are mainly characterized by realtime processing capabilities in order to react 
immediately to user actions and by close integration of humans by novel output and interaction devices. Realized 
applications either locus on the imitation of real worlds (interior design, architecture, landscapes) or the creation of 
imaginary, abstract worlds with even different behaviour (experimental research, art). 

Today many virtual reality applications concentrate on the output side on the generation of a sufficiently high 
image frame rate. This approach was motivated by the availability of high performance graphics workstations, 
adequate rendering techniques and the stimulation of one primar human sense - our visual perception capability. 

Within the last years low-cost personal computer and mid-size workstations have been equipped with audio 
devices originally for multimedia purposes. On the other hand electronic music devices like synthesizers have 
made rapid progress in sound generation and usability especially with the introduction of digital components and 
the midi programming and controlling interface. 

Although audio hardware and software is readily available today, few systems use acoustic sensations to enhance 
the man-machine communication in virtual reality. Acoustical simulations - in certain aspects similar to visual 
light simulations - contribute a great deal to a realistic impression of virtual worlds. The algorithms presented can 
either be applied in realtime or they can precompute acoustical characteristics which will be evaluated later in 
realtime. 
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The virtual reality system "Virtual Design" of IGD concentrates on the imitation of natural environments, i.e. 
offices, buildings, cities, landscapes. It is completely data-driven, i.e. the scene with all visual and aural objects and 
associated attributes is defined in a datafile-interface. 

This paper introduces the facilities at the VR demonstration centre of IGD in Darmstadt and the architecture of the 
VR system "Virtual Design". The available qualities of acoustical simulations are explained and an overview over 
the specification of worlds with the datafile-interface is given. Furthermore the interpretation and processing of the 
audiovisual objects is outlined. Several examples demonstrate the usage of the system and the achievable 
audiovisual effects. 


2 THE VIRTUAL REALITY DEMONSTRATION CENTRE OF IGD - HARDWARE 
FACILITIES 

Since the beginning of 1993 the Fraunhofer Society has established a demonstration centre for virtual reality partly 
located in Darmstadt. The intention was to provide and establish a facility where different systems can be 
demonstrated and evaluated, where consulting, know-how- transfer and training is offered. 

The demonstration centre in Darmstadt provides a high quality environment of virtual reality hardware (figure 1) 
to run different applications. Central node is a high performance graphics workstation (SGI RE), to which various 
input devices (dataglove, flying joystick, spaceball, spacemouse, 3d tracking system) and output devices (head- 
mounted-display, large screen stereoscopic projection, video) are connected. 

The graphics workstation is linked to a multimedia-workstation and a personal computer, which controls the 
midi-equipment (synthesizer and effect processor), the 3d audio processing card and a power unit control card. 
The audio output signals of both machines are combined in a mixer, which also includes the signals of a compact 
disk player, a digital audio tape, a microphone and a video player. The mixer drives speakers as well as the 
headphones in the head-mounted-display via the 3d audio card. 

There are many different processing tasks to solve in virtual reality applications: the simulation of world 
behaviour, the computation of the visual and acoustic representation, the management and access of peripheral 
devices and the interpretation of user actions. As there is no integrated machine available which satisfies all needs, 
specialized computing nodes distributed in a net offer optimal use of processing power. Besides the high 
performance graphics workstation a compute server (Convex) for simulation purposes and a multimedia 
workstation (SGI Indigo), 3d audio processing hardware (focal point) and midi-equipment (Akai SIOOO, Alesis 
QuadraVerb) for sound rendering are available. 

To manage and utilize the full processing power of a heterogenous net a distributed system is required. In Virtual 
Design a sound server addresses multimedia workstations as well as midi-devices via a common programming 
interface. The server stores a table of digitally sampled sounds, which can be manipulated. 

The multimedia-workstation permits calculation (e.g. the convolution of impulse responses with dry sound 
samples) and subsequent rendering of digital sounds plus basic manipulation (volume). The midi-equipment 
consists of a synthesizer and an effect processor. The synthesizer permits the realtime manipulation of volume, 
frequency and stereo level. The effect processor has a number of controllers to simulate the reverberation of a 
sound in an enclosure. Adjustable parameters of the effect processor are: 
enclosure type (room, hall, echo chamber), 
decay time, 

degree of diffusion (number of reflections), 
reverberation density (density of impulse response), 
low frequency decay. 
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high frequency decay. 

The devices installed at the demonstration centre allow the presentation of virtual worlds to a large audience via 
stereoscopic large screen projection and speakers as well as an individual self-experience via head-mounted- 
display and headphones (also simultaneous). 



Figure 1 Facilities at IGD's demonstration centre in Darmstadt 
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QUALITIES OF ACOUSTICAL RENDERING 


To animate virtual worlds with sound which have been designed as models of our real world, realistic sounds of 
our environment have to be integrated. Common environmental sounds are discrete (steps on the floor, door 
closing etc.) and continuous sounds (teapot boiling, drilling etc.) caused by material interaction (sound events), 
vibrating objects (machine parts), liquids (splashes), aerodynamic sounds (exploding balloon) or sounds from 
loudspeakers (radio, tv). Sounds are also used for alarms (fire alarm), warnings (r.p.m. of engine getting too high) 
and messages (engine going). Sounds are damped through objects (windows), they can "crawl" around objects 
(pillar, especially low frequencies) and the reflections are mostly specular (due to wavelength). A frequency shift 
can be noticed from passing objects (police car with siren). Humans can detect the position of a sound source in 
space (azimuth better than elevation). 

To satisfy at least some of the above mentioned acoustic effects Virtual Design offers several acoustic rendering 
qualities. The room acoustical simulation algorithms have many parallels to visual simulation algorithms, the main 
differences caused by the different propagation speed and wavelength of sound and light and the time resolution. 

Sound events: An object is assigned an acoustic material which has a reference to a sound source. Ideally a matrix 
of sounds should be stored containing all possible collision sounds between moving objects. In the toolkit this is 
reduced to a list for the cursor (fingertip of dataglove echo) and all collision-sensitive objects. A sound source 
indexes a sound stored in the sound server. The toolkit cycles an action loop to update image frames; in each cycle 
collisions are tested and a callback function is eventually evoked. When an object has an audio event attribute the 
referenced sound source is triggered. Examples show a drum kit which can be played with the dataglove and an 
assembly /disassembly system (3d puzzle). 

Direct propagation: In our daily life three acoustic parameters can be noticed when sound propagates directly 
from a sound source to a receiver: volume damping (caused by distance and medium), frequency shift for moving 
objects (doppler effect) and position in space. If a sound source, a receiver and an object have been assigned direct 
propagation attributes, these objects are passed for computation of the audio parameters to the algorithm. Then 
the parameters are passed for auralization to the sound server. An example shows a beetle proceeding over a 
plain. 

Statistical approach: This approach is based on assumptions stated by W.C. Sabine. The algorithm computes an 
average reverberation time independent of receiver and sound source position. Thus it is possible to characterize a 
room type, e.g. a living room, a conference room or a church hall. The computed parameter value configures the 
effect processor, which performs the reverberation in realtime. Whenever an alternative acoustic model is chosen, 
the reverberation value is recalculated. An example shows a conference room with different reverberation 
characteristics. 

Room Acoustics: Room acoustical simulations compute the energy distribution of an energy pulse emitted from a 
sound source and detected by a receiver within an enclosure. The simulation considers the exact geometry with 
reflections, absorption, dispersion and air turbulences (wind). The receiver detects the so-called impulse response 
(figure 2) which can be convoluted with any dry sound sample. 

The image source algorithm mirrors actual sound sources at enclosing faces, which can be executed quite easily for 
rectangular rooms. In Virtual Design the user navigates to the desired listening position, then the impulse response 
is calculated at this position, visualized and sent to the audio server for convolution. The original, dry sound and 
the convoluted sound can both be rendered to compare the results. 

Ray tracing algorithms trace the propagation of sound particles or rays in an arbitrary world. They are similar to 
visual simulation algorithms in respect to ray propagation and intersection calculation. 

Database: The acoustic Tenderer require additional objects like sound sources, receivers and acoustic material 
attributes. Visual and acoustic Tenderer are in many respects very similar; they both operate on geometrically 
defined scenes. It is the different propagation speed of light and sound which causes algorithmic differences. The 
Tenderer compute parameters which are used by an audio server to manipulate digitally sampled sounds. 
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The visual model comprises polygonal objects, lights, camera and a scene description with multiple object 
references. Additional acoustic objects supplement the visual model: 

- medium (propagation speed of sound, damping), 

- acoustic material (local reflections, sound), 

- sound source, 

- listener (can be coupled with the observer), 

- impulse response, 

- sound sample format. 

The medium and material data are held in a table with some predefined entries like air medium or carpet material. 
Objects have visual as well as acoustic attributes; they can be classified to be processed by the desired Tenderer. A 
datafile-interface (see chapter 4) provides a convenient definition of the visual as well as the acoustic scene model. 
When all model description data is loaded into the database, a cyclic rendering process (see chapter 5) is started. In 
a single cycle each object is inspected, the attributes are evaluated and the object is passed to the specified visual or 
acoustic Tenderer for further processing. Typically all visual objects are rendered and acoustic parameters are 
computed which will alter e.g. a continuous sound. 


4 THE DATAFILE INTERFACE 

An application processed by Virtual Design is defined by a set of environment variables and data files. The 
environment variables specify device ports, output screen size and device configuration. The model and 
interaction modes are defined in various data files: 

Control file: References all other data files (root file). Moreover a lot of parameters can be specified to define the 
coordinate system, background colour, logo, dataglove representation and tracking parameters, alternative object 
representations, collision-sensitive objects, images, menu configuration and appearance. Functions can be assigned 
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to menu entries like fly mode, converter level, automatic camera recording, toggling of system states, triggering of 
sound sources and acoustic rendering algorithms. 

Scene file: References and places all objects in the scene, sets objects attributes and transformations, camera and 
lights. 

Visual definition files: Defines the object geometries (set of polygons), attributes, colours and textures. 

Acoustic definition file: Defines receivers and sound sources with attributes, acoustic object properties and 
configures the audio server. 

Camera file: Defines a set of specifications for the camera (observer), can be used to load and replay prerecorded 
walkthroughs. 

Animation file: Defines a set of transformation matrices for objects. With these data objects can be animated in a 
world (the beetle for instance). 

To integrate geometric models into Virtual Design a number of CAD formats of various modelers including 
Autodesk's AutoCad are supported. 



Figure 3 Snapshot of a walkthrough in interior design (CAD data by Asplan, postprocessing by CAD & Art) 
Virtual Design provides a convenient and powerful development platform for virtual reality applications (figure 

3 ). 


Virtual Design is an example for a system with abilites corresponding to aspects specified near the end of the axis 
of the DIP-model [EAFF-93]. At the heart of the virtual reality toolkit of IGD a central database is located. Many 
modules with different tasks operate on the database and manipulate database objects (figure 4). 

Objects can be attributed as collision-sensitive.These objects are tested each frame against a cursor object 
(forefinger of dataglove object for instance). Whenever a collision is detected, a callback function is evaluated to 
perform a predefined action [Zach-92], 

The tracking system transforms device-specific position and orientation data in space into a world coordinate 
system, which can be interpreted by the navigation module. This module computes the transformation matrices 
for the observer object and interaction objects driven by devices like the dataglove [FeFG-93]. 

Human hand gestures detected by dataglove sensors are compared against a table of predefined gestures. Upon 
recognition the appropriate action function is evoked [Wirt-92], 

A radiosity system can be used to precalculate colour values of polygon patches. An integrated ray tracer 
computes photorealistic images, which can be shown as still frames during a walkthrough as an example for high 
quality rendering [MiiUG-93]. The radiosity system features also level-of-detail object hierarchies, visibility 
culling, context-sensitive object transformation and time management of frame display rates. It serves as a testbed 
for further enhancements of the toolkit. 

Standard modelers like AutoCad can be used to create and update a scene model. A number of CAD object 
formats can be imported and loaded into the database to define the scene geometry and the visual objects with 
their attributes. Additional objects like a hand model and a 3d menu object are included by default for visual 
feedback of interaction. 

A sophisticated rendering system provides an object database interpreted by several visual and acoustic Tenderer 
with realtime processing capabilities. The object database comprises up to now for visualization a two level 
hierarchy: For modelling purposes the objects are grouped into sets, for rendering purposes the objects are 
transformed into a flat, referenceless but fast interpretable cache structure. To master very large and complex 
scenes visibility culling, level-of-detail scene analysis and a hierarchical database is currently under development. 
The visual Tenderer uses SGI's GL to achieve the necessary performance. 

The acoustic Tenderer require additional objects like sound sources, receivers and acoustic material attributes. The 
rendering system currently comprises four different acoustic Tenderer [AsGo-93]. The Tenderer compute 
parameters which are used by an audio server to manipulate digitally sampled sounds. 

A programmable video-recorder integrates motion pictures into virtual reality applications. A peripheral device 
controller steers AC-powered electric devices like fans to create the illusion of movement. 

With these modules a sample virtual reality system "Virtual Design" has been realized, which permits the 
processing of highly differing applications. Virtual Design is extremely flexible and can be configured through a 
data interface. Possible interactions are specified with a configurable 3d menu. 
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The basic structure of Virtual Design in respect of acoustic rendering is as follows: 


main: 


load system configuration 
load scene 

initialize peripheral devices (interaction device, tracker, audio server) 

if (any object with statistic attribute) 
evoke statistic Tenderer 
send reverberation value to audio server 

navigation loop 

get navigation data 
render next frame 

if (any objects with direct attribute) 

send play (object->source, volume, frequency, 3d position) to audio server 

if (menu evoked) 

disable navigation 
evaluate selected menu item: 

image source Tenderer: 

evoke image source Tenderer 

visualize impulse response 

send impulse response to audio server 

trigger sound source: 

send play (source) to audio server 

trigger convoluted sound: 

send play convoluted sound to audio server 

enable navigation 

if (collision detected) 

if (detected object has event attribute) 

send play (object->source) to audio server 


6 Examples 

The acoustic rendering qualities described in chapter 3 are demonstrated in examplary virtual world applications, 
which can be seen on the supplemental video tape. Some snapshots from these virtual world explorations are 
shown below. Each application is generated by loading and interpreting file data for the visual and acoustic scene. 

In figure 5 an example for the application of sound events is pictured: a drum set placed in the middle of a virtual 
room can be played with the dataglove. 
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Figure 6 shows an example of an assembly/disassembly system with sound feedback. Notifying sounds are 
releasen when a part is grabbed, when two parts collide and when they fit together and finally an applause 
rewards the lucky user. 


Figure 7 pictures a beetle standing in the middle of a plain. When the simulation is switched on, the beetle passes 
several times the observer (equals listener). Change of volume, frequency shift and stereo position can be noticed. 



Figure 7 Passing beetle Figure B Image source algorithm 

Figure 8 shows a snapshot from an audiovisual virtual world, where the visual model is shown in wire frame 
quality, the acoustic model in solid representation with absorption attribute mapped onto a blue colour scale. In 
the center of the model the calculated impulse response is visualized; the vectors depict direction and detected 
energy of a sound source, the colour depicts the arrival time on a red (early) to blue (late) colour range. 

All examples have been realized with predefined scenes modeled with standard CAD packages. Acoustic objects 
with attributes (receivers, sound sources) and geometry-related attributes as absorption coefficients have been 
added in the acoustic datafile. 


7 Conclusion and Future Work 

The integration of audio features in virtual reality applications contributes a great deal to a natural simulation of 
imaginary worlds. It leads a step further to the ultimate goal to stimulate all human senses and let humans totally 
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immerse in virtual "worlds. With the Virtual Design system of IGD a variety of audiovisual worlds are easily 
defined and presented for exploration to the user. 

Standard CAD packages and animation systems allow the specification of scenes with objects, geometry, visual 
attributes, camera parameters and light sources; they do not support the interactive modeling of acoustic object 
attributes, receivers and sound sources. The extension of available modelers with acoustic related functions will be 
one area of research in future. Furthermore the integration of a particle tracer, a geometry simplification module 
and the connection and application of appropriate signal processing hardware for binaural convolution of sounds 
in realtime will be evaluated. 
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